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Abstract — We investigate a reconstruction limit of compressed 
sensing for a reconstruction sclieme based on the Li-norm 
minimization utilizing a correlated compression matrix with 
a statistical mechanics method. We focus on the compression 
matrix modeled as the Kronecker-type random matrix studied in 
research on multiple-input multiple-output wireless communica- 
tion systems. We found that strong one-dimensional correlations 
between expansion bases of original information slightly degrade 
reconstruction performance. 

I. Introduction 

A novel approach of data compression, termed compressed 
sensing (CS), has recently been drawing great attention. The 
central assumption of CS is the sparsity of original informa- 
tion, which seems plausible for many real world signals. For 
exploiting this property, much effort has been paid in both 
research directions of theory and application [l]-[3]. 

The basic idea of CS is summarized in the following linear 
equation: 
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(1) 



(Throughout this article vectors and matrices are denoted in 
bold letters), x'^ E denotes A^-dimensional coefficient 
vector of the original information / G expanded over 
the basis e (l < i < N), namely / = J^t^i^t^^^ 
and y e is a P-dimensional vector, which describes 
compressed information available from P times observations 
with observation vector xpi, yi = f ■ ^pi (1 < i < P). F 
is a P-by-A^ compression matrix whose element is given by 
Fij — xpi ■ 4>j. In this article the compression matrix F is 
regarded as random, which means that we are dealing with 
random expansion bases and random observation vectors. The 
compression rate is defined by a = P/N < 1. The original 
coefficient a;° is sparse and modeled by the distribution, 

P{xf) = (1 - p)5{x^) + pexp (-(x°)V2) /V2^, (2) 

that is, p represents the density of non-zero coefficients. Under 
the above setting, ii-norm minimization offers an appropriate 
feasible algorithm for reconstruction of the original coefficient 
(termed Li-norm reconstruction), 

minimize ||a;||i subject to y{= Fx'^) = Fx, (3) 



where ||a;||p= limg^+o X^i Nj I^^^- The remaining problem is 
whether the solution x coincides with the original coefficient 
ccq. We can expect that below a certain critical value of the 
compression rate ac, the original coefficient x^ cannot be re- 
produced even if we make use of the ii-norm reconstruction. 
The aim of this article is to evaluate this critical value ac in 
the limit of P, iV — > oo (and a — const.) utilizing a statistical 
mechanics method. 

By the way, this problem is quite similar to the perfor- 
mance evaluation problem of linear vector channels in wireless 
communication, and accordingly the analysis scheme with 
the statistical mechanical approach for code-division multiple- 
access (CDMA) or multiple-input mukiple-output (MIMO) 
communication [4]-[6] can be applied in the limit P, A^ — > oo. 
Kabashima et al. have already investigated the performance of 
the Li-norm reconstruction (to be precise general Lp-norm, 
though p < 1 is the reconstructable case) using a statistical 
mechanical method in a basic scenario in [7], where F is 
composed of independently and identically distributed (i.i.d.) 
random variables, and evaluated the reconstruction limit ac- 
The evaluation value accords with the one that has been 
assessed in [8], [9] using combinatorial geometry methods. 
(As a related study, noisy compressed sensing was investigated 
using replica method in [10], for which perfect reconstruction 
is not possible as long as the noise intensity is not negligible). 

In this article, as a second step of the investigation, we 
consider a more advanced case in which F is provided as 

F = /R^H/r;, (4) 

and our goal is to evaluate critical value ac for such F. Here, 
i?!- and Rt are a P- and an A^-dimensional square symmetric 
matrix, respectively. The square root of a square matrix A is 
defined as, A ~ \fA \fA. H is a random P-by-A^ rectangular 
matrix whose elements are i.i.d. Gaussian random variables 
of zero mean and variance A^^^. This random matrix H 
effectively implies a situation in that the expansion bases and 
the observation vectors are statistically uncorrected. In this 
modeling, the matrices i2t and i?i represent the correlations 
among the expansion bases 4> and those among the observation 
vectors i/j, respectively. Random matrix of this type, F, is 
known as the channel matrix in the Kronecker model of the 



MIMO communication system, whose performance is investi- 
gated by Hatabu et al. [6] with a statistical mechanical scheme. 
Accordingly, by application of this method it is expected that 
the reconstruction limit of the ip-norm reconstruction can also 
be estimated. In the subsequent sections we explain the details 
of the analysis. 

II. Replica analysis 

In this section we describe the outline of the analysis. As 
we mentioned, the analysis is based on that for the Kronecker 
channel in the MIMO communication system [6], and the 
details of the analysis are also discussed in this work. 

Following the discussions in [7], let us first define the cost 
function of the ip-norm reconstruction using the quenched 
average of free energy, which is a standard technique for 
deaUng with a random system in statistical mechanics, 

= - lim lim |- lim -^ln[Z"{/3,y)]p^,o, (5) 

where [-jFja;" denotes the average over the random matrix F 
and the original coefficient with the distribution (2). We 
also define the replicated partition function .Z"(/3, y) for n e 
N as 
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(6) 



From these expressions, we easily see that the cost function 
Cp is nothing but the minimized norm with the constraint 
y = Fx = Fx^ for given y. After performing the average 
over the random matrix H in F we have 
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The function Gs'^HrS is defined as 

GstrMA) = j dXpR^{X)\u(^I-^Ay (9) 

The function pr^ {\) in the definition of G^t jf^s is the 
eigenvalue distribution of the matrix R^. 

Assuming repUca symmetry, let q = Qab {a 7^ b), 
Q = Qaa, = Qao, and u = Qoo- Here, u is defined 
as w EE N^^ JYl^dx'^P{x°)x°^RtX° = ^TiRf From 
these assumptions it follows that Saa = Q — 2m + u and 
Sab = q — 2m + u (a b). By diagonalization of the matrix 
S, we can evaluate the G^tj^^'s -dependent part. 
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From the saddle-point method, the a;-dependent part, including 
the Lp-norm and the constraint, is expressed as. 
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(11) 



where {S)ab = Qab - Qao - Qob + Qoo and Q is an n- 
dimensional matrix defined by the constraint, 

n(")(Q) = n / '^^^ 1 n Hx^^Rt^" - NQaa) \ 

X I JJ Six'^'^Rtx'' - NQab) \ I n '5(a="'^i2tx° - NQ^o) \ 

U<6 J U=l J 

X {^(x°^i2ta;° - iVgoo)}exp (-f^P ||a;«||p) . (8) 



where the interaction between replicas is removed by intro- 
ducing auxiliary variable z (Hubbard-Stratonovich transforma- 
tion). For simpler expression of Cp, let us define new variables 

m = p-^m, X = p-^q, X = - q), Q = P'^-'^Q + q) 
and the function 

(/>p(h, Q)=-^ jii^^o™'^'" I ^^^^^ ~ h'^^/Rix+ \\x\\p+e > ■ 

(12) 

For /3 — > cxD, the a;°-dependent part is rewritten as 

n J P{x°i)dx° J Dze^Y>{-mncl>p{fh^/R,x'' + ^/^z,Q)) . 

(13) 

Combining these results and inserting the expression of the 
function G^tr^s and its derivative in the limit r +0, we 
have the final expression of the cost function 



= Extr 

Q,'m,x,Q,fh,x 



a(q- 2ni + u) I XX qQ , 
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(14) 



The cost function depends only on the correlation matrix 
-Rt between expansion bases of the original information, 
and does not depend on the matrix between observation 
vectors, which indicates that the observation procedure is not 
essential for CS. This seems reasonable because sparsity of 
the original coefficient is significant and observation is not for 
CS. Accordingly, we must concentrate only on the effect of 
the correlation matrix Rt on the Lp-norm reconstruction. 

As mentioned above, Cp is nothing but the minimized Lp- 
norm, and the expression Eq. (14) tells us that the minimized 
Lp-norm is given by the solution of the extremization problem. 
The remaining problem is whether the original coefficient 
is correctly reconstructed typically from the solution of the 
extremization problem. Remembering the fact that x is the 
result of the reconstruction and cc" is the original coefficient, 
from Eq.(8) q — m{— u) must hold when the reconstruction is 
successful. Therefore, the scheme for finding the reconstruc- 
tion limit etc is as follows: vary the parameter a. (compression 
rate) and p (density of non-zero coefficients), then solve 
the extremization problem, and examine whether the solution 
satisfies q = m{— u). 

We have completed the repUca analysis as above, and the 
cost function (14) we obtained describes the information of the 
Lp-norm reconstruction for arbitrary correlation matrices R^ 
and ilf (As you see R^ will eventually become irrelevant). 
However, one problem remains: the cost function Cp includes 
the function (j)p{h,Q), which is defined by the minimization 
problem with A'' variables, whose expression is not so simple 
(For Li-norm this problem can be solved numerically in 
principle because the minimization function is unimodal). For- 
tunately, for certain classes of Rt, this minimization problem 
can be expressed relatively simply, which allows us to evaluate 
the reconstruction limit in a tractable manner. In the case 
without correlation ilt = we see that the minimization 
problem with iV variables is reduced to the problem with 
a single variable, and the cost function changes to the one 
obtained in [7]. In the following section, we wiU give another 
simple but nontrivial example, for which the minimization 
problem is numerically tractable. 

III. example: adjacent correlation 

Let us consider that the correlation matrix R^ has a tridi- 
agonal form, as discussed in [6], in the context of the MIMO 
communication system, defined by 
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(15) 
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This corresponds to the case that only adjacent matrix com- 
ponents (or adjacent expansion bases) have correlation. This 
matrix can be decomposed as ilt = V^t (Cholesky 



decomposition with boundary term), where 
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Here l± = (^1 + 2r ± y/l - 2r)/2. In what follows we 
focus on the Li-norm minimization. In the present case 
u = TrRt/N = p holds, and the cost function is rewritten as 



Ci = Extr I ^ + ^ _ ^ + TOTO 

+ n / p^'^y^ j n ^^^■'^1 ^) [ ' (17) 



where hi = m{l+x^ + l-x^+i) + s/x^i and x^ is defined 
periodically as = a;°. (For simplicity we denote h = 

{hi, • • • , /lAf}). The function (f)i{h, Q) in the cost function Ci 
can be transformed as 



bi{h,Q) = ^mm^\^x'^RtX-h^^/R^x+\\x\\l 



i i i ) 



(18) 



The variables x, h are also defined periodically, xq = xn, 
xn+1 = xi, ho = hM, and hN+i = h\. x^ is given by the 
solution of the minimization problem, namely for each i 

-^Mh, Q) = {Qx* - mx°) + r{Qx*_i - fhxti) 

+r{Qx*+i - rhx°i+i) - ^/^{l+Zi + l-Zi-i) + sgn{x*) 
= (19) 

is satisfied. As seen above, the minimization problem for 
each i includes only variables with three sequential indices 
i — and i + 1, which indicates that the minimization 
problem is on a one-dimensional chain. It should be noted 
that the minimization function is unimodal, and sequential 
minimization for each variable enables us to find the minimum 
when we try to search it numerically. The computational cost 
of this procedure is 0{N) and feasible. 

The extremization condition of the cost function can be 
expressed using the solution of the minimization problem. 



denoted by a;*, 

- ^ a _ aiq - 2m + p) 

Q = m = -, x = — 2 ■■ 

X X 



^ = -ir^ n / D~zMP{x% z^{l+x* + l_x*^,). 

(20) 

Remember that x* depends on Q, m, -^/x, Zi, through Eq. 
(19). 

The next issue is the reconstruction Hmit as discussed in 
[7]. As mentioned before, if the Li-norm reconstruction works 
successfully, q = m(= p) holds and the right hand side of 
the equation for £ in Eq. (20) vanishes. The extremization 
conditions for X)9;"^ can be combined by using a new 
variable Xi = x^ — x^, 



X = 



X'N' 



W / Dhdx^lP{x'l)d. 



x^ Xj {xj +rxj-i+rxj+i ) 



(21) 

From this expression it follows that Xi = (namely x* = x^) 
is obtained from the extremization conditions as a solution 
of successful reconstruction. On the other hand, by inserting 
fh = Q = a/x into Eq. (19), 

d , p:. a ^ \ 

■—(t)i{h,Q) = - [Xi + rxi-i + rxi+i) 
dx*i X 

-\f%l^Zi + l-Zi-i) + sgn{xt + Xi) = 0. (22) 

This equation indicates that in the Umit x Xj should 
vanish faster than 0(x) in order for the solution = to 
exist. (In this case x is 0(1)). From the insight above, we 
rescale the variable as a; — ^ {x/cx)x. The remaining equations 
in terms of x, X, Xi are 

^ = ^ n / + ^^j--^ + rxj+i), 

i j 

i^-^^^J^Ii J Dzidx°P{x°)Y^ Zj{l+xj+l.xj+i)j, 



X = X 

_d_ 
dx 



-(j)i{h,Q) = {xi +rxj_i +rxi+i) 



-^/2{l+Zi + l-Zi-i) + sgn (^Xi + x°) = 0. (23) 

The second equation has two solutions: x = and the factor 
in the bracket is unity. The first solution x = corresponds 
to successful reconstruction, Xi = x* (note that x is rescaled), 
and the second (which satisfies x 7^ 0) amounts to unsuccess- 
ful reconstruction. The equation for the threshold is obtained 
by inserting x = to the second solution. In conjunction with 
the remaining two equations, we finally have the equations 
for the reconstruction limit (Note that in the evaluation of the 



reconstruction limit, we must perform multiple integrals in the 
equations for x and x> and we can use the Monte Carlo method 
in numerical evaluation). 

Equation (23) is the main result. By the procedure men- 
tioned above, we can estimate the reconstructions ac as 
the function of p (density of non-zero coefficients) and r 
(correlation parameter in the compression matrix). For Rt = I 
(without correlation), we can recover the reconstruction Umit 
obtained in [7] directly from the expression of Eq. (23). 

IV. EVALUATION OF RECONSTRUCTION LIMIT 

For the adjacent correlation discussed above, we estimate 
the reconstruction limit by using the result of the replica anal- 
ysis in Eq. (23). In Fig. 1, the dependences of reconstruction 
limit ac on the density p are shown for uncorrelated (r = 0) 
and correlated (r = 0.5) cases. The difference between two 
results are very small over aU region of p, which indicates 
the effect of adjacent correlation is very small. In Fig. 2 
the dependence on the correlation parameter r is depicted 
for p = 0.5. In the case without correlation, Kabashima et 
al. obtained ac = 0.8312... for p = 0.5 [7]. In the region 
of small r, we cannot observe the deviation of ac from the 
uncorrelated case r = 0. On the other hand, for the strongly- 
correlated case r = 0.5, we observe a slight increase in ac, 
which implies that strong correlation worsens the performance 
of the Li-norm reconstruction. For r = 0.5, ac is estimated as 
ac = 0.84057(14), indicating that the performance falls about 
1% from r = in terms of the reconstruction limit. 

For verification of the results from replica analysis, we 
also conducted a numerical experiment of the ii-norm re- 
construction. We used the convex optimization package for 
MATLAB developed in [11], [12]. The results are shown in 
Fig. 3. The dependence of ac on the dimension of the original 
coefficient A'' is shown. For comparison with replica analysis, 
we also performed scaUng analysis with quadratic function 
regression, and estimated the value of ac for A'' ^ oo Umit. 
The results give ac = 0.84017(28) for N ^ oo, which clearly 
indicates the increase in the value of ac (or degradation of the 
reconstruction performance) as expected. The reconstruction 
limit estimated by extrapolation is very close to the one from 
replica analysis, which enforces the validity of the result from 
repUca analysis. 

V. CONCLUSIONS AND DISCUSSION 

We investigated the performance of the Lp-norm (especially 
Li-norm) reconstruction with a correlated compression matrix 
by repUca analysis. We obtained the expression of the cost 
function for the Kronecker-type compression matrix for gen- 
eral correlation matrices Rt and R^. 

The noteworthy issues in the results are summarized as 
follows. First, the cost function does not depend on the correla- 
tion matrix R, describing the correlation between observation 
vectors, which indicates that the observation procedure is not 
significant in CS. This can also be understood from the fact 
that we can eliminate the correlation matrix R^ by redefinition 
(or rotation) of the random matrix H. 
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Fig. 1. Reconstruction limit etc as a function of p. We compare tlie cases 
r = 0.5 (x) and r = (+, uncorrelated). We set tlie dimension of tlie 
original coefficient as TV = 10®. For the evaluation of the multiple integrals 
in Eq. (23), we perform the average over 100 samples of z, tc'^. The difference 
between two results is very small. 
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Fig. 2. Dependence of reconstruction limit etc on correlation parameter r. 
We set p = 0.5 and the dimension of the original coefficient as N = 10®. For 
the evaluation of the multiple integrals in Eq. (23), we perform the average 
over 100 samples of z,a;". In the vicinity of r = 0.5, the deviation of Oc 
from the value for r = is clearly observed. (For p = 0.5, the reconstruction 
limit is estimated as Oc = 0.8312... [7].) For r < 0.4 the deviation is not 
visible and the result is not shown in this figure. At r = 0.5, the criticality 
is estimated as Oc = 0.84057(14). 
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Fig. 3. Results of Li-norm reconstruction experiment. We set p = 0.5 
and r = 0.5. For each N (=dimension of the original information) we 
took the average over 10^ samples. The broken line indicates the scaling 
by quadratic function regression. For A'^ — > oo, we clearly see the increase 
in the reconstruction limit from the case of no correlation (ac = 0.8312... 
for p = 0.5 [7]), which is consistent with the result from replica analysis. 
The value at A'^ oo from the extrapolation is ac = 0.84017(28), which 
is very close to the value from replica analysis. 



Second, we considered the correlated compression matrix, 
whereas the original coefficient is uncorrelated. However, our 
result can be reinterpreted as the case of a correlated original 
coefficient and uncorrelated expansion bases by redefinition 
(or rotation) of these quantities. 

Finally, we found that the performance of the Li-norm 
reconstruction is robust against the small correlation between 
adjacent expansion bases of the original coefficient. By in- 
corporating the strong correlation, the reconstruction perfor- 
mance slightly falls. This is a quite natural result because 
the correlation between expansion bases implies the loss of 
the original information by redundant bases, which makes 
the reconstruction much more difficult. We should also keep 
in mind that we discussed the reconstruction limit only in 
the case that the correlation matrix is tridiagonal, and there 
is a possibility that another correlation matrix might highly 
degrade the performance (note that our analysis offers the 
result for general correlation), which will be for future work. 
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